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Abstract 



For Poisson distribution Pois{n, A) with A ^ 1, n ^ 1 we propose to determine 

significance as S = — Th.e significance S coincides up to sign with often 

V A 

used significance. For experiments which measure the same quantities the natural 
not unique i 

Scomb{Sl-,S2) 



but not unique rule for significance combining is 

SiUi + S2<T2 



where a\ and are variations. We also propose the rule for significances combining 
for the case with systematic errors. 
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"Suppose one experiment sees a 3-sigma effect and another sees a 4-sigma effect. What 
is combined significance?" [1] . 

In this note we discuss the problem of significances combining on the example of 

Poisson distribution^. Namely for Poisson distribution Pois{n, A) with A ^ 1, ?^ ^ 1 we 

propose to determine significance as 5* = ^o&g ^ rpj^g significance S coincides up to 

V A 

sign with often used significance. For experiments which measure the same quantities the 
natural but not unique rule for significance combining is 



where cxi and (72 are variations. We also propose the rule for significances combining for 
systematic errors. 

Suppose the CMS experiment detects in July 2010 10300 events with the expectation 
10000 events and in august it detects 9700 events with the expectation 10000 events. The 
probability to detect n events is determined by Poisson formula 

Poisin,\) = —e-^. (1) 

For A ^ 1, n ^ 1 Poisson distribution is approximated by normal distribution with 
mean /j, = X and variance a = \/\. 

For A ^ 1, Hobs ^ 1 the often used significance S is determined by the approximate 
formula 



(2) 

V A 

For our example we find the July and august CMS significances 



_ 110300 - 100001 _ 

yioooo " ' ^ ^ 



_ 19700 - 100001 _ 

^august — I „„„„ — V^} 



^loooo 



^See also [2, 3] 
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If we deal with collection of CMS data for July plus august we find that data are 
described by Poisson distribution with Xjuiy+august = >^juiy + Kugust and Uobsjuiy+august = 
nobsjuiy + 'nobs,august- In accordancc with the formula (2) we find that July + august 
significance is 

o \'^obs,july ~l~ '^obs.aiifj 11.^1 ■^july ^august] /r\ 

ijjuly+august — / = U y'^ ) 

y Xjuly ~l~ ^august 

in perfect agreement with the theory expectations. This trivial example illustrates the 
fact that it is impossible to combine only significances, i.e. 

S july+august 7^ -^("SjuZj/) Saugust) (6) 

One of the possible solutions is to determine significance as 

^^ riobs-X 
V A 

The definition (7) coincides with often used significance definition up to sign. For 
the case of events excess it is positive and for the opposite case it is negative. For the 
definition (7) the proposed rule for significances combining is 



Ocomb{Oi, b2) - , - 1 {^) 

V Ai + A2 ^(jI + al 

The rule (8) is in fact trivial generalization of Stouffer's method [4] and for Poisson 
statistics it looks rather natural^. The generalization of the asymptotic formula (8) to the 
case of not big Ai and A2 is straightforward. 

Other lesson from our example and formula (8) is that the significance combining not 
necessary leads to increase of the significance. 

Note that the situation changes if we investigate the case when the parameter A in 
Poisson distribution is not known. For our example we find that \juiy = 10300, Xaugust = 



^The rule (8) is the realization of the fact that the sum of Poisson processes with Ai and A2 is the 
Poisson process with A = Ai + A2. 
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9700 and Xjuiy+august — 20000. In the assumption that Xjuiy — ^august the analysis of 
july+august data gives Xjuiy = Xaugust = 10000 

Other example is the case when CMS detects in July 10300 events with the expectation 
10000 and in august it detects 10100 events with the expectation 10000. 

Again we find 



10300 - 10000 

Vmoo 



_ 10100 - 10000 _ 

" ^7iool~ ~ ' ^ ^ 

And for CMS data for july+august we find 

tijuly+august - ^^qooq " ^ ~ ^'') 

It is not difficult to take into account systematic effects related with nonexact knowl- 
edge of the parameter A in Poisson distribution (1). Namely, suppose theoretical uncer- 
tainty in the A parameter calculation is eA. For such uncertainty the generalization of the 
formula (7) is 



g _ nobs — A 
" VA+(6A)2' 
The generahzation of the formula (8) looks 



(12) 



. ^iA + (eA0^ + ^,VA, + (6A,)^ 

/Ai + As + (e(Ai + \2)Y 



Note that combining significances for experiments with different cuts in general does 
not help in search for new physics. Really suppose we look for the number of isolated 
muons with transverse momentum: 

(a) . 100 GeV < pr < 200 GeV, 

(b) . pt > 200 GeV, 

(c) . Pt > 100 GeV. 



4 



It is evident that the combination of the (a) "experiment" with the (b) "experiment" 
is equivalent to the (c) "experiment". And it is not clear which experiment (a), (b) or 
(c) will give the biggest significance (the biggest evidence in favour of new physics). The 
details will depend on the tested model of new physics. 

This work was supported by RFBR grant N 10-02-00468. 
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